Papers with RL algorithm
Reinforcement Learning with Token-level Feedback for Controllable Text Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for controllable text generation are guided by coarse-grained feedback, which may lead to suboptimal performance owing to semantic twists or progressions within sentences. |
| Approach: | They propose a reinforcement learning algorithm which formulates TOken-LEvel rewards for controllable text generation and employs a "first-quantize-then-noise" paradigm to enhance the robustness of the RL algorithm. |
| Outcome: | The proposed algorithm can achieve superior performance on single-attribute and multi-attract control tasks. |
AdaptThink: Reasoning Models Can Learn When to Think (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large reasoning models have demonstrated remarkable capabilities in tackling complex tasks. |
| Approach: | They propose an algorithm to teach reasoning models to choose the optimal thinking mode based on problem difficulty. |
| Outcome: | The proposed algorithm reduces the average response length and improves accuracy on three math datasets. |
Invocation Refiner: A Plug-and-Play Module for Rectifying LLM Tool Invocations (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in Tool-Integrated Reasoning (TIR) however, the practical application is often hindered by frequent errors in tool invocations, such as incorrect tool names, invalid parameters, wrong tool-call order, or malformed invocation formats. |
| Approach: | They propose a specialized post-processing module that performs independent reasoning on the input of a frozen upstream LLM and an advanced RL algorithm to improve the tool-use reliability of base LLMs. |
| Outcome: | The proposed module improves task completion rates and invocation accuracy over the raw outputs of various upstream LLMs on a diverse set of tool-use and reasoning benchmarks. |
KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Task-oriented dialogs (TOD) require a model to generate a response that optimizes for task-related metrics. |
| Approach: | They propose a faster generation procedure that samples from independent next-word distributions and introduce a fine-grained reward function to help the model focus on learning key information in a dialog. |
| Outcome: | The proposed algorithm achieves state-of-the-art performance on an offline task with 15% training time reduction compared to a standard RL algorithm using auto-regressive generation. |
AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin (2026.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) has emerged as an effective approach for enhancing the reasoning capabilities of large language models. |
| Approach: | They propose an algorithm that optimizes cross-entropy loss using advantages enhanced through a margin-based estimation scheme. |
| Outcome: | Experimental results show that AAPO improves group relative advantage estimation compared to other methods. |
Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion (2024.emnlp-main)
Copied to clipboard
Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Azar, Olivier Pietquin, Matthieu Geist
| Challenge: | Reinforcement Learning (RL) is a method used to fine tune Large Language Models (LLMs) using a reward model trained from preference data to better align with human judgment. |
| Approach: | They propose a Reinforcement Learning (RL) algorithm that can estimate the optimal policy even from off-policy data. |
| Outcome: | The proposed algorithm can estimate the optimal policy even from off-policy data. |
Generative Reward Modeling via Synthetic Criteria Preference Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Generative Reward Models (GenRMs) leverage synthesized Chains of Thought (CoT) but this approach introduces risks of overoptimization due to the inability to guarantee the correctness of the CoTs. |
| Approach: | They propose a criteria-based preference tree for GenRMs that uses chain of thought to generate reasoning . they show that synthesized data can be learned using a long CoT format . |
| Outcome: | The proposed model shows significant improvements over baselines on multiple human preference benchmarks. |
EMTIR-GRPO: Efficient Multi-Tool Augmented Large Language Models via Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Tool-integrated reasoning (TIR) enables large language models to invoke external tools for tasks beyond their internal capacity but often suffers from tool overuse. |
| Approach: | They propose an algorithm that uses a composite reward to model tool costs and tool efficiency. |
| Outcome: | The proposed algorithm models heterogeneous tool costs and encourages more cost-effective tool-use strategies. |
Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Evidence-Augmented Policy Optimization (EAPO) improves long-context reasoning performance . Xu et al., 2025): large language models are a critical part of NLP . |
| Approach: | They propose an Evidence-Augmented Reasoning paradigm that uses a group-relative reward to improve evidence quality. |
| Outcome: | EAPO significantly improves long-context reasoning performance compared to baselines. |
The Retrieval Bottleneck: Scaling Laws for Reinforcement Learning in RAG (2026.acl-long)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) has become the dominant paradigm for building knowledge-intensive language systems. |
| Approach: | They propose a sigmoidal scaling law that shows that retrieval quality determines the asymptotic performance ceiling. |
| Outcome: | The proposed model achieves strong performance on knowledge-intensive benchmarks while retaining the predictable scaling long available for pre-training but previously absent in RAG-RL. |
Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work studies RLVR through token entropy, arguing that high-entropies drive exploration and should receive stronger updates. |
| Approach: | They propose a correctness-aware reinforcement framework that performs fine-grained advantage modulation over low-entropy segments. |
| Outcome: | The proposed framework improves accuracy over strong RL baselines across three backbones and six math benchmarks while maintaining high-entropy exploration. |
Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization (2026.acl-long)
Copied to clipboard
Yifeng Ding, Hung Le, Songyang Han, Kangrui Ruan, Zhenghui Jin, Varun Kumar, Zijian Wang, Anoop Deoras
| Challenge: | Current reinforcement learning methods suffer from coarse-grained, trajectory-level rewards that provide insufficient learning signals for complex multi-turn interactions, leading to training stagnation. |
| Approach: | They propose a novel RL algorithm for training large language models for multi-turn tool-integrated reasoning (TIR) that incorporates three innovations: turn-level reward assignment that provides fine-grained feedback for individual turns, return-based advantage estimation where normalized discounted returns are calculated as advantages, and self-supervised reward shaping that exploits self-supervision signals from generated code to densify sparse binary outcome-based rewards. |
| Outcome: | The proposed algorithm outperforms GRPO by 3.0% across diverse math reasoning benchmarks and improves grepo by 3.9% on commonsense reasoning and program synthesis tasks. |